Vehicle image processing apparatus and vehicle image processing method

ABSTRACT

A vehicle image processing apparatus includes a group of cameras, a drawing unit that converts a captured image into an image viewed along a line of sight running from a predetermined position in a predetermined direction, a viewing-line-of-sight changing unit that detects whether a first line of sight is switched to a second line of sight, and a viewing-line-of-sight generation/updating unit that acquires parameters concerning the first and second lines of sight after detecting the switching and generates a parameter which is gradually changed from the parameter of the first line of sight to the parameter of the second line of sight. Moreover, the drawing unit generates, on the basis of the gradually changed parameter, an image which is gradually changed from an image viewed along the first line of sight to an image viewed along the second line of sight.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.14/703,112 filed May 4, 2015, which is a continuation of U.S.application Ser. No. 12/907,314 filed Oct. 19, 2010, which is based uponand claims the benefit of priority of the prior continuationapplication, filed under 35 U.S.C. §111(a), of PCT Application No.PCT/JP 2009/055003, filed Mar. 16, 2009, and claims priority to JapanesePatent Application No. 2008-140931, filed May 29, 2008, the disclosuresof which are herein incorporated in their entirety by reference.

FIELD

The present invention relates to a vehicle image processing apparatusused to visually check an environment around a driver's vehicle when thecar is running.

BACKGROUND

In order for a driver of a vehicle to visually check an environmentaround the driver's vehicle, there is a system that allows the driver toobjectively and intuitively grasp the situation around the driver'svehicle by combining and converting a plurality of vehicle-mountedcamera images into an image (overhead image) that views with a virtualline of sight that is a line of sight extending, for example, from thesky above the driver's vehicle to the driver's vehicle and displayingthe image. An extended version disclosed is, for example, an imagegeneration device that: turns a captured vehicle-mounted camera imageinto a three-dimensional space model without change; maps colorinformation of the image onto the three-dimensional space model;converts the three-dimensional space model into an image seen from anarbitrary visual point; and displays the image.

If a three-dimensional shape (the shape of a cylinder, bowl orquasi-cylinder) made up of curved and flat surfaces is used as a modelon which a camera image is projected just as the image generation devicedoes, the advantage is that it is possible to view not only an overheadimage around a vehicle but also all surrounding images including the skyat the same time.

What is also disclosed for such an image generation device is atechnique of helping a driver check the safety in driving a vehicle bychanging the images being displayed for the driver depending on therunning state of the vehicle, i.e. the running speed, the steering angleof the wheel and the state of the results of detection by an objectdetection sensor.

As a result, it is possible to change images to be displayed that aresupplied from a vehicle-mounted camera at a viewing position determinedby a scene such as the operation of a vehicle.

The followings are disclosed as conventional techniques.

-   [Patent Document 1] Japanese Patent No. 3,286,306-   [Patent Document 2] Japanese Patent No. 3,300,334

SUMMARY

A vehicle image processing apparatus comprising: a distortion correctionunit that acquires a captured image which is an image that has capturedan area around a vehicle with the use of at least one camera; a drawingunit that uses at least one predetermined projection shape to convertthe captured image into an image viewed along a virtual line of sightthat is a line of sight running from a predetermined position in apredetermined direction; a viewing-line-of-sight changing unit thatdetects whether a first virtual line of sight which is a predeterminedvirtual line of sight is switched to a second virtual line of sightwhich is a different virtual line of sight from the first virtual lineof sight; and a viewing-line-of-sight generation/updating unit thatacquires at least one type of parameter value concerning the virtuallines of sight each for the first and second virtual lines of sightafter detecting that the first virtual line of sight is switched to thesecond virtual line of sight, and generating a parameter value that isgradually changed from the parameter value of the first virtual line ofsight to the parameter value of the second virtual line of sight;wherein the drawing unit generates, on the basis of the graduallychanged parameter value, at least one changed image which is an imagethat is gradually changed from a first image which is an image viewedalong the first virtual line of sight to a second image which is animage viewed along the second virtual line of sight, wherein theviewing-line-of-sight changing unit detects that the first virtual lineof sight is switched to the second virtual line of sight if theswitching is the switching of a virtual line of sight which isregistered in advance or if the difference between the unchanged andchanged virtual lines of sight is greater than or equal to a specifiedvalue.

The object and advantages of the invention will be realized and attainedby means of the elements and combinations particularly pointed out inthe claims.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory and arenot restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an example of the configurationof a vehicle image processing device according to an embodiment of thepresent invention;

FIG. 2 is a flowchart illustrating an example of a process of thevehicle image processing device according to the embodiment of thepresent invention;

FIG. 3 is a flowchart (a drawing process and a display process)illustrating an example of a process of the vehicle image processingdevice according to the embodiment of the present invention;

FIG. 4 is a diagram illustrating an example of an interpolation method(linear interpolation) of a parameter of a viewing line of sightaccording to the embodiment of the present invention;

FIG. 5 is a diagram illustrating an example of an interpolation method(B-spline function) of a parameter of a viewing line of sight accordingto the embodiment of the present invention;

FIG. 6 is a diagram illustrating an example of an interpolation method(ellipse's arc) of a parameter of a viewing line of sight according tothe embodiment of the present invention;

FIG. 7 is a diagram illustrating an example of a line-of-sight movementpath from a midair overhead image to a vehicle forward image accordingto the embodiment of the present invention;

FIG. 8 is a diagram illustrating an example of an image and shape thatis display data of a related object (a driver's vehicle) according tothe embodiment of the present invention;

FIG. 9 is a diagram illustrating an example of an image and shape thatis display data of a related object (an obstacle and the like) accordingto the embodiment of the present invention;

FIG. 10 is a diagram illustrating a sample image for explaining aprojection method that uses a projection map according to the embodimentof the present invention;

FIG. 11 is a diagram for explaining a projection method that uses aprojection map of each camera according to the embodiment of the presentinvention (combining of images);

FIG. 12 is a diagram for explaining a projection method that uses aprojection map used for all captured images according to the embodimentof the present invention;

FIG. 13 is a diagram illustrating an example of a projection imageconsisting of a spheroidal surface and a plane representing a roadsurface according to the embodiment of the present invention;

FIG. 14 is a diagram illustrating a model space that is ato-be-processed area of a related object shape setting unit according tothe embodiment of the present invention;

FIG. 15 is a schematic diagram illustrating a driver's vehicle and anobstacle according to the embodiment of the present invention;

FIG. 16 is a diagram illustrating a related object image and thedisposition of the related object image according to the embodiment ofthe present invention;

FIG. 17 is a schematic diagram illustrating the position of a viewingline of sight and a viewing direction, as well as a diagram illustratingan example of an image in each viewing line of sight;

FIG. 18 is a schematic diagram illustrating an example of each image andan example of a line of sight concerning each image in order to explainthe difference between a conventional forward image and pseudo driver'simage; and

FIG. 19 is a diagram illustrating a list where usage determinations of apseudo driver's line of sight are listed.

DESCRIPTION OF EMBODIMENT

According to the above conventional techniques, when a viewing line ofsight (a virtual line of sight) is changed, an image viewed by a driverabruptly changes. Therefore, it is difficult for the driver to promptlygrasp the situation around a vehicle that the image depicts after theline of sight is changed.

The above problem will be further described with reference to FIG. 17.In FIG. 17(a), as an example of changing the viewing line of sight, anoverhead view seen from the sky above a driver's vehicle and across-sectional view seen from the left side of the driver's vehicleillustrate a position (predetermined position) of the viewing ling ofsight and a viewing direction (predetermined direction). FIG. 17(b) isan example of images for each viewing line of sight illustrated in FIG.17(a). For example, an image with a viewing line of sight A couldsuddenly change to an image with a viewing line of sight D. In thismanner, when an image suddenly changes to another with a largedifference in the field of view as illustrated in FIG. 17(b) where theviewing positions and the viewing directions are different, it isdifficult for a driver to promptly determine where an attention area ofthe previous image is positioned in the succeeding image after thechanging.

To solve the above problems, the embodiment is to provide a vehicleimage processing device capable of changing a viewing line of sightsmoothly.

FIG. 1 illustrates an example of the configuration of a vehicle imageprocessing device (a vehicle image processing apparatus) according to anembodiment of the present invention. A vehicle image processing device100 is equipped with a driving information acquisition unit 1, a roadinformation acquisition unit 2, a direct specification unit 3, aviewing-line-of-sight changing unit 4, a viewing-line-of-sightgeneration/updating unit 5, a drawing method determination unit 6, arelated object determination unit 7, a group of cameras 8 (The group ofcameras 8 is made up of a plurality of vehicle-mounted cameras), adistortion correction unit 9, a drawing unit 20 and a display unit 16.The drawing unit 20 is equipped with a projection shape setting unit 10,a projection image conversion unit 11, a related object shape settingunit 12, a related object image setting unit 13, a projection conversionunit 14, and an image superimposing unit 15.

According to the present embodiment, volatile hardware resources, suchas a CPU 200, memory and hard disk drive, a nonvolatile hardwareresource, which is a storage medium 201, and software resources storedin the storage medium 201 work closely with each other to realize eachof the above units.

Incidentally, according to the present embodiment, an example ofvehicle-mounted cameras mounted on a driver's vehicle that act as thegroup of cameras 8 will be described for simplicity of explanation.However, the group of cameras may include cameras installed on anyplaces other than the driver's vehicle, such as infrastructure camerasinstalled on roads or other vehicle cameras; through a communicationmeans such as wireless communication, captured images supplied from theabove cameras may be acquired as input images for use. Even when anycameras other than those mounted on the driver's vehicle are used, theoperation is described in the same way as the vehicle-mounted camerasmounted on the driver's vehicle. Only when particular attention needs tobe paid because the operations and processes are different, thedifference will be described if preferable.

The following describes the operation of the vehicle image processingdevice 100 based on each of the above units.

FIGS. 2 and 3 are flowcharts illustrating an example of the operation ofthe vehicle image processing device 100.

First, the distortion correction unit 9 reads a captured image from eachof the corresponding cameras of the group of cameras 8 (Step S1) andcorrects the distortion of the captured image to obtain a correctedimage (Step S2). The distortion is attributable to lenses of the camerasand the like and is an already-known distortion. The distortioncorrection unit 9 therefore uses an already-known method to correct.Incidentally, the correction of the image's distortion is performed atthe beginning for simplicity of explanation. However, the correction ofthe image's distortion may be performed when the projection imageconversion unit 11 converts the image as described below or when theprojection conversion unit 14 projects the image. Or alternatively, thecorrection of the image's distortion may be omitted.

When the correction of the distortion is performed by the projectionimage conversion unit 11, it is possible to carry out the correctionwith a simpler process if the correction of the distortion is regardedas adjusting pixel positions inside the captured images and if, when aprojection map is created that illustrates the relationships betweenpixel positions inside the captured image that is to be referred to andpixel positions inside the converted image, the adjustment of pixelpositions for correcting distortion is included in advance. Similarly,when the correction of the distortion is performed by the projectionconversion unit 14, it is easy to carry out the correction if theadjustment of pixel positions for correcting distortion is included inthe relationships between pixel positions inside the captured image,which are used in the projection conversion unit 14 to acquire the colorof the pixel of each point of a projection shape, and the coordinates ofcharacteristic points of a projection shape.

Hereinafter, a captured image is considered to be a distortion-correctedcaptured image when preferable.

Here, suppose that when a captured image of a camera mounted anywhereother than the driver's vehicle is used, a determination is made as towhether there is a camera used for the reading of the captured image inthe surrounding area before the captured image is read, and cameraparameters, such as the position and direction of the camera, and thecaptured image are obtained through a wireless means or the like at anytime. Incidentally, the camera parameters may not be acquired directly:a list of camera parameters corresponding to a camera ID may be acquiredin advance, and only the camera ID may be acquired and converted toenable the camera parameters to be acquired when preferable. In thiscase, the correction of the distortion may be performed in the driver'svehicle after the captured image is read out. However, it is notefficient to acquire parameters for correcting the distortion from acamera mounted on another vehicle or infrastructure cameras installed onthe roadside through wireless communication every time. Therefore, it isdesirable that the distortion-corrected captured image be read out toomit the correction in the driver's vehicle or that the correction becarried out by converting and acquiring the correction parameters fromthe above list with the use of the ID.

Then, the direct specification unit 3 makes a determination as towhether there are the contents (about customization of the vehicle imageprocessing device 100 by a user) directly specified by a user, such as adriver, who uses the vehicle image processing device 100 (Step S3). Ifthere are the directly-specified contents (Step S3, YES), the specifiedcontents are acquired (Step S4). A more detailed description of thespecified contents will be given along with a description of thecontents of how a viewing line of sight is changed. Incidentally, it isnot preferable for the processes of steps S3 and S4 to be performed herein the flowchart. If there is no specific problem in consistency, thevehicle image processing device 100 may perform the process as apriority interrupt process after detecting the specifying at any time.Incidentally, the direct specification unit 3 is primarily aimed atallowing a user to correct or customize the contents when preferablethat are automatically determined inside the vehicle image processingdevice 100. Therefore, the direct specification unit 3 may be omitted.

Then, the driving information acquisition unit 1 makes a determinationas to whether the driving information has changed, while the roadinformation acquisition unit 2 makes a determination as to whether theroad information has changed (Step S5). If there is a change (Step S5,YES), both the driving information for the driver's vehicle and the roadinformation are acquired (Step S6). Here, the driving informationcontains: the speed and travelling direction of the driver's vehicle,which can be acquired form a vehicle speed sensor, gyroscope, gear orthe like; and the driving operation (information about the driving ofthe vehicle) concerning a forward movement, left or right turn andbackward movement, which can be acquired from the operation or amount ofoperation of a wheel, gear, direction indicator or the like. The roadinformation is the information that can be acquired from a wirelesscommunication device that carries out car-navigation or communicationbetween vehicles on the road. The road information contains the shapesof roads, the types of roads, the types of urban districts, the state ofcongestion, road information, and map information of stores and the like(information about the road around the vehicle) and is acquired from amap database, GPS or the like along with the position of the driver'svehicle. Incidentally, the shape of a road and an obstacle may bedirectly acquired by the driver's vehicle with the use of an externalsensor such as a millimeter wave sensor or laser sensor. Moreover, eachpiece of the information acquired by the driving information acquisitionunit 1 and the road information acquisition unit 2, particularly theinformation acquired by the road information acquisition unit 2, ismainly referred to in the process of identifying a scene to change theviewing line of sight as described below or in the process of making adetermination as to whether a related object is used, an arrangementmethod or the like. Therefore, the driving information acquisition unit1 and the road information acquisition unit 2 may be omitted.

Then, the viewing-line-of-sight changing unit 4 makes a determination asto (or detects) whether the current viewing line of sight (a firstvirtual line of sight) has changed to another viewing line of sight (asecond virtual line of sight) on the basis of the acquired drivinginformation, road information, or contents of a user's instruction (StepS7). If there is the change (Step S7, YES), the viewing-line-of-sightchanging unit 4 proceeds to a process of the viewing-line-of-sightgeneration/updating unit 5 (Step S8). If there is no change (Step S7,NO), the viewing-line-of-sight changing unit 4 proceeds to an updatedetermination process of the viewing line of sight (To step S11).Incidentally, in a broad sense, the changing of the viewing line ofsight can be divided into two: the change resulting from the updating ofline-of-sight parameters associated with the calculation ofline-of-sight interpolation by the viewing-line-of-sightgeneration/updating unit 5 as described below; and the change resultingfrom the changing of the lines of sight in response to mainly thechanging of scenes. Hereinafter, for the purpose of convenience, theformer is referred to as the updating of the viewing line of sight, andthe latter as the changing of the viewing line of sight. In the lattercase where the viewing line of sight is changed, images generated fromthe unchanged and changed viewing lines of sight are a first image and asecond image, respectively.

The changing of the viewing line of sight is the changing of one viewingline of sight to another in response to each usage scene that is set inadvance or the changing of a viewing line of sight to an arbitrary onethat a user directly specifies. A determination rule and the contents ofchanging may be used to make a determination as to whether to perform analready-known changing process.

With reference to the above FIG. 17, an example of changing a viewingline of sight depending on scenes will be described. As described above,FIG. 17(a) is a schematic diagram illustrating the positionalrelationship between the viewing line of sight and the driver's vehicle.FIG. 17(b) illustrates a converted image when actually viewed along theviewing line of sight illustrated in FIG. 17(a). According to thepresent embodiment, the viewing line of sight is basically set relativeto the driver's vehicle for simplicity of explanation. Suppose that theviewing line of sight moves in conjunction with the movement of thedriver's vehicle unless otherwise stated.

For example, for the viewing lines of sight, the following relationshipsare set in advance: the viewing line of sight A of FIG. 17 that islinked to the driver's vehicle moving forward; the viewing line of sightB that is linked to the driver's vehicle making a right turn; theviewing line of sight D that is linked to the driver's vehicle snakingthrough a narrow streat; the viewing line of sight C that is linked tothe driver's vehicle moving backward in a parking lot or the like. Forthe scenes and viewing lines of sight that are preset in the abovemanner, the viewing-line-of-sight changing unit 4 makes a determinationas to whether the current driving information, road information or thecontents of a user's instruction matches the preset scenes as well aswhether it is preferable to change the viewing line of sight. Forexample, the viewing-line-of-sight changing unit 4 detects that thedriver's vehicle moving forward is to make a right turn on the basis ofthe direction indicator and determines that the current viewing line ofsight A for the scene of forward movement is replaced by the viewingline of sight B for a right turn.

Incidentally, the example illustrated in FIG. 17 is an example in whichthe viewing lines of sight that are set and linked in advance to thescenes are used without change. However, fine adjustments may be made tothe viewing lines of sight that are linked to the actual, precisesituations of each scene before the viewing lines of sight are used. Forexample, the following process is considered available: It is decidedonly to change the direction of the viewing line of sight and the zoom(focal distance) in order to enable an enlarged image of a detectedobstacle to be viewed; particular parameters, such as the direction ofthe viewing line of sight and visual point positions, are adjusted inaccordance with the automatically detected position of the obstacle,direction and the like; and a viewing line of sight is replaced by theabove viewing line of sight. Incidentally, the example of determiningwhether the changing of the viewing line of sight is preferable is oneexample; it is possible to determine to change the viewing line of sighton the basis of other scenes and conditions. For example, as thechanging of the viewing line of sight resulting from the detection of ahigh-priority obstacle as a user's instruction or safe drivinginformation, an enlarged image of the obstacle may be displayed in thefield of view or alternatively, the viewing line of sight may change tothe one seen from the sky above the driver's vehicle to focus on thepositional relationship with the obstacle.

There may be a determination rule that the viewing-line-of-sightchanging unit 4 changes the viewing line of sight when the degree ofdifference between the unchanged and changed virtual lines of sight isgreater than or equal to a specified value (for example, when the visualpoint position and the viewing direction are greater than or equal to aspecified distance and angle).

When it is determined that during the process of updating a give viewingline of sight the viewing line of sight is changed to a new one, thevehicle image processing device 100 stops the ongoing updating of theviewing line of sight and starts changing to a new viewing line ofsight. In this case, the first image of the unchanged image in theprocess of changing to a new viewing line of sight does not have aviewing line of sight corresponding to some scene; the first image is achanged image in the preceding process of interpolation updating that isgenerated from the viewing line of sight for the determination ofchanging that is a viewing line of sight generated by the interpolationof line-of-sight parameters described below.

In general, it is preferable that the viewing line of sight mostsuitable for the current scene be used to display. Therefore, theprocess of interpolation updating is stopped. However, the process maygo on without being stopped. In this case, the process is the same aswhen it is determined that the viewing line of sight is not changed(Step S7, NO).

Incidentally, for the purpose of convenience, the process ofinterpolation updating is stopped in the above description. However,fine corrections may be made to the contents of interpolationcalculation so that the changing to a newly specified viewing line ofsight is included, and the updating of the viewing line of sight maycontinue. In this case, making fine corrections to the contents ofinterpolation calculation is regarded as determining the new contents ofinterpolation calculation, and the subsequent processes (after YES ofstep S7) are performed.

By the way, when it is difficult to figure out from where a new viewingline of sight is seen, it may be impossible to promptly figure out inwhich direction the line of sight is seeing even if a changed image fromthe current viewing line of sight is used. To avoid the above, aspecified viewing line of sight, for example a line of sight that isclose to that of the naked eye of a driver (referred to as a pseudodriver's line of sight, hereinafter), may be always regarded as a firstvirtual line of sight, and a changed image whose viewing line of sightis always varying may be created from the above line of sight.Therefore, it is possible to view the changed image whose line of sightis changed in a pseudo manner from the current line of sight, which issubstantially equal to that of the naked eye of a driver, to the viewingline of sight suitable for a scene. Thus, it is possible to figure outwhat and from which direction the final viewing line of sight is seeingin a more concrete way.

FIG. 18 is an explanatory diagram illustrating an example of the pseudodriver's line of sight as well as a difference between the pseudodriver's line of sight and the forward viewing line of sight and forwardimage used in a conventional technique. FIG. 18(A) illustrates anexample of a conventional forward image (forward image A) as well as anexample of a pseudo driver's image (pseudo driver's image B) used in thepresent embodiment. FIGS. 18 (B), 18(C) and 18(D) schematicallyillustrate an example of a viewing area that can be viewed with the useof line-of-sight parameters, such as the visual point position of theline of sight concerning each of the images, the angle of field and theline-of-sight direction, and the lines of sight.

The conventional forward image is substantially the same as avehicle-mounted image that is generated by taking a picture in a forwarddirection that is actually mounted on a recent vehicle. FIG. 18(A)illustrates a forward captured image just as a forward image. For theforward captured image, in order to take a wide-range picture without ablind spot, a wide angle lens of about 180 degrees is used for takingthe picture. The picture turns out to be a distorted spherical image asa whole, like the conventional forward image A of FIG. 18(A).Practically, in order to make the image easier to watch, instead ofusing the forward captured image without change, an image from which thespherical distortion has been removed is used as a forward image in manycases. However, only the distortion of the subject in the image ischanged; there is no large change in the line-of-sight parameters andimage-capturing area, i.e. in a spatial domain that can be viewed.Therefore, for simplicity of explanation, the forward captured image isillustrated here without change.

As illustrated in FIG. 18(A), the line-of-sight parameters and viewingarea of the pseudo driver's image B are significantly different from theforward image A. As described above, the main purpose of the forwardimage A is to help a driver in driving. Accordingly, a picture isgenerally taken and displayed with a wide viewing angle of field in boththe vertical and horizontal directions so as to contain a blind spotthat the driver may not see at the present time, such as a nearby bottomportion of the driver's vehicle; the forward image A and a forward lineof sight are a line of sight that is based on such image-capturingparameters. For example, in the bottom portion of the forward image A ofFIG. 18(A), the nearby bottom portion of the driver's vehicle isdisplayed. Meanwhile, the pseudo driver's image B is aimed atreproducing a scene that the driver is now seeing with the naked eye.Therefore, it is not preferable to cover a blind spot; given a field ofview at the time of close observation, the angle of field may be aviewing angle of about 50 to 60 degrees, which is substantially the sameas that of a typical image-capturing camera. Accordingly, the nearbybottom portion of the driver's vehicle is not displayed in the pseudodriver's image B of FIG. 18(A). However, the pseudo driver's image B isan easy-to-see image having a larger image area that the driver isactually viewing, with the surrounding situation as a whole squeezedmore than the forward image A. FIG. 18(B) schematically illustrates aviewing pyramid depicting the viewing line of sight and viewing areathat generate two images of FIG. 18(A) as a typical four-sided pyramid(View volume; the base is spherical) with a visual point positioned atthe vertex. In the diagram, the viewing pyramid of the pseudo driver'simage B is entirely inside the viewing pyramid of the forward image A.However, the viewing pyramid of the pseudo driver's image B may not beentirely enveloped in the viewing pyramid of the forward image A. Inthis case, for a projecting portion of the viewing area, an image of acamera other than a forward image-capturing camera may be used.

Unlike the forward line of sight of the forward image A, as illustratedin FIGS. 18(C) and 18(D), the position of a pseudo driver's line ofsight that is the line of sight of the pseudo driver's image B and aline-of-sight direction vector are not preferably on the central axis ofthe driver's vehicle and in the traveling direction of the driver'svehicle, respectively. For the forward line of sight of A, a visualpoint is typically positioned at the center of the driver's vehicle thatis a representative point of the driver's vehicle, i.e. close to theforward tip portion on the central axis of the driver's vehicle, withthe line-of-sight vector in the direction of the central axis ordirection straight ahead, which is the traveling direction of thedriver's vehicle. Meanwhile, the center of the pseudo driver's line ofsight of B is positioned not at the central axis of the driver's vehiclebut at where the driver actually sits in the vehicle. That is, thecenter is positioned to the left or right from the central axis of thedriver's vehicle depending on where the wheel is attached. The positionof the naked eye is determined by detecting an amount of operation at atime when the lengthwise position of the driver's seat, the angle of thebackrest and the position of the headrest are adjusted. As illustratedin FIG. 18(A), The line-of-sight vector is set in the direction in whichthe driver is likely to pay the most attention to the center of thefacing-and-sneaking-through lane when for example waiting to make aright turn (Incidentally, whether the driver's vehicle is waiting tomake a right turn is for example determined by the current speed of thedriver's vehicle and whether the direction indicator is operated). It isnot preferable for the line-of-sight vector to be in the same directionin which the driver's vehicle travels. When such a pseudo driver's lineof sight is used, as illustrated in FIG. 18(A), the positionalrelationship of each subject relative to the center of the imageindicated by “+” in the forward image A is different from that in thepseudo driver's image B.

In that manner, compared with a conventional forward image, the processof determining the pseudo driver's line of sight that more closelyrepresents that of the driver's naked eye and creating the changed imagein which the pseudo driver's line of sight serves as a virtual line ofsight is effective when the line of sight along which the driver iswatching is significantly different from a new line of sight to whichthe line of sight is to be switched.

Incidentally, the pseudo driver's line of sight is not preferablylimited to one. An arbitrary number of pseudo driver's lines of sightmay be prepared with varying line-of-sight parameters, including thevisual point position that varies according to the sitting height of thedriver, the viewing angle that varies according to the age of thedriver, and the close observation direction, or line-of-sight direction,that is different from the traveling direction of the driver's vehicledepending on whether a right or left turn is made; the pseudo driver'sline of sight may be switched to another when preferable in accordancewith a driver's instruction, the surrounding driving environment, thedriving state of the driver's vehicle, driver's information and thelike. In addition, the above values may be calculated for use so as tomore precisely reflect the current state each time. Or alternatively,prepared values may be corrected for use. For example, by detecting thesitting height of the driver, the position of the face and the like witha dedicated camera in the vehicle or various sensors, the pseudodriver's line of sight may be more precisely customized for use.

The pseudo driver's lines of sight may be used not only when a changedimage to an arbitrary viewing line of sight as a first virtual line ofsight is created but also when the current viewing line of sight istemporarily changed to the pseudo driver's line of sight in order tofigure out in which direction an arbitrary viewing line of sight isseeing; and can be used for either the first or second virtual line ofsight before or after the interpolation. When an arbitrary viewing lineof sight is used for the first or second virtual line of sight, theline-of-sight parameters of the pseudo driver's line of sight may beused during the process of line-of-sight interpolation described belowso as to go through the pseudo driver's line of sight in the middle ofthe changed image that uses the above.

Whether the pseudo driver's line of sight is used or not may bearbitrarily determined. For example, the determination may be made whenpreferable on the basis of the magnitude of a difference between theline-of-sight parameter of the pseudo driver's line of sight, or thenaked eye of the driver, and that of the current viewing line of sightor to-be-displayed viewing line of sight or on the basis of how urgentit is to change the viewing line of sight, which is estimated from thedriver's vehicle and the surrounding environment or the state ofstopping the updating of the viewing line of sight; and the pseudodriver's line of sight may be used. FIG. 19 illustrates an example ofthe usage determination.

FIG. 19 is a list where examples of the usage determination of thepseudo driver's line of sight are listed and the conditions for usingthe pseudo driver's line of sight are listed as one example. When eachitem agrees with the contents described in a condition, the pseudodriver's line of sight is used according to the usage contents.Incidentally, what is illustrated in FIG. 19 is one example. The usagedetermination may be made with another arbitrary condition. The pseudodriver's line of sight may be used in a different manner from what aredescribed in the table.

Returning to the explanation of the flowchart of FIG. 2, theviewing-line-of-sight generation/updating unit 5 then makes adetermination as to whether to perform the interpolation of the viewingline of sight from the contents of how the viewing line of sight ischanged (Step S8). When the interpolation is performed (Step S8, YES),the contents of how to interpolate are determined (Step S9). As a methodof making a determination as to whether to perform the interpolation ofthe viewing line of sight, the viewing-line-of-sight generation/updatingunit 5 for example determines to perform the interpolation aftercalculating the difference of each parameter (the visual point position,viewing direction, viewing angle (angle of field), or focal distance(zoom factor)) of the unchanged and changed viewing lines of sight andthen regarding the case where the difference is greater than or equal toa threshold value as a large change of the viewing line of sight.Incidentally, if the changing of the viewing line of sight is only thechanging between the viewing lines of sight that are registered inadvice, whether the interpolation is performed in advance or not may bedetermined according to a change pattern of which registered viewingline of sight is changed to which registered viewing line of sight, andthe viewing-line-of-sight generation/updating unit 5 may determine toperform the interpolation when a change pattern is the above changepattern. In this case, the viewing-line-of-sight generation/updatingunit 5 can determine whether to interpolate just by checking the changepattern without calculating the difference of a parameter of theunchanged and changed viewing lines of sight every time.

When the interpolation of the viewing line of sight is not performed(Step S8, NO) as a result of the determination as to whether to performthe above interpolation (Step S8), the viewing-line-of-sightgeneration/updating unit 5 simply changes the viewing line of sight(Step S10) and proceeds to a determination process of the drawing method(Step S13).

Then, the viewing-line-of-sight generation/updating unit 5 confirmswhether the updating is still preferable or to be stopped in order tomake the final viewing line of sight (second virtual line of sight) inaccordance with the contents of how to interpolate (Step S11). If theupdating is not to be stopped and is still preferable (Step S11, YES), aprocess of performing the updating of the viewing line of sight byreferring to the contents of how to interpolate and changing the viewingline of sight little by little is repeatedly performed (Step S12). Ifthe updating is stopped because the final viewing line of sight isobtained (Step S11, NO), the viewing-line-of-sight generation/updatingunit 5 does not update the viewing line of sight and the processproceeds to the subsequent determination process of the drawing method(Step S13).

Incidentally, in the flowchart, for simplicity of explanation, thecontents of how to perform the interpolation of the viewing line ofsight are determined in advance, and the viewing line of sight isupdated by referring to the contents at any time. However, the procedureis not limited to the above. The process of determining the contents ofhow to perform the interpolation of the viewing line of sight and theprocess of updating the viewing line of sight may be performed at thesame time, and the interpolation calculation of the viewing line ofsight may be performed every time the viewing line of sight is updated.Or alternatively, a part of the contents of how to perform theinterpolation of the viewing line of sight, which is for example aninterpolation method concerning which calculation formula is used in theinterpolation of the viewing line of sight, may be determined, and thecalculation of the viewing line of sight may be performed in a concretemanner when the viewing line of sight is updated.

The interpolation method of the viewing line of sight of the presentembodiment adopts a publicly-known, arbitrary interpolation calculationprocess in which parameter values of the unchanged and changed changesmoothly. For example, the interpolation process in which the value ofthe visual point position of the viewing line of sight is used as aparameter is determined by the method illustrated in FIGS. 4 to 6.

In FIG. 4, the viewing line of sight A is schematically illustrated asthe unchanged viewing line of sight, and the viewing line of sight B asthe changed viewing line of sight, when seen from the left side of thedriver's vehicle in the case of the cross-sectional view (FIG. 4(a)),and from the sky above the driver's vehicle in the case of the overheadview (FIG. 4(b)). FIG. 4 illustrates an example of performing a simplelinear interpolation process of parameters. That is, as a path of thevisual point position interpolation of the unchanged viewing line ofsight A and the changed viewing line of sight B, a segment AB connectingthe visual point position A to the visual point position B is used, andthe interpolation is performed in such a way that the line-of-sightposition gradually moves and changes on the interpolation path from theviewing line of sight A to the viewing line of sight B. Incidentally,the line-of-sight position may not move and change at constant speedacross the interpolation path. The interpolation updating of theline-of-sight position may be performed at arbitrary speed: for example,the moving and changing speed is faster around the viewing line of sightA and the viewing line of sight B and slower in any other places. Thespeed can be set at any value using the traveling interval and travelingdistance on the interpolation path. If an interpolation updating pointis updated based on the movement of the line-of-sight position thatmoves at regular intervals on the interpolation path, the interpolationupdating of the position of the viewing line of sight can be performedat constant speed. If the traveling distance is set smaller than othersby changing the traveling interval, the interpolation updating of theposition of the viewing line of sight can be performed at slower speedin the portion than in other portions.

FIG. 5 illustrates an example of an interpolation process that uses analready-know B-spline function to recursively calculate a curve fromboth two endpoints of the visual point position A of the viewing line ofsight A and of the visual point position B of the viewing line of sightB and one control point (calculation control point). One control pointdefines the shape of a curve as well as how a plane containing the curveis arranged in a three-dimensional space.

With the main focus on variations of the arrangement of thethree-dimensional position, the following three patterns are illustratedin FIG. 5 as examples: the case where a segment connecting a controlreference point O related to the driver's vehicle to a point (midpointM) on the segment AB is used (FIG. 5(a)); the case where the point(midpoint M) on the segment AB and a viewing-line-of-sight-directionvector on the point are used (FIG. 5(b)); and the case where the point(midpoint M) on the segment AB, a line perpendicular to the segment ABthat passes through the point, and a viewing-line-of-sight-directionvector on A or B are used (FIG. 5(c)). The only difference between thethree patterns is the calculation method for the calculation controlpoint P; the same method of performing B-spline interpolation from thecalculation control point P, the visual point position A and the visualpoint position B is used. Thus, the following describes the calculationmethod of the interpolation position by mainly referring to FIG. 5(a).

In the example of FIG. 5(a), except for the line-of-sight positions(visual point positions A and B) that are endpoints, theviewing-line-of-sight generation/updating unit 5 calculates thecalculation control point P needed to calculate a position interpolationpath as a point that is away from the midpoint M by a distance that isin a predetermined ratio to the length of the line AB on the linepassing through the midpoint M of the segment AB and the controlreference point O where the center of gravity of the driver's vehicle islocated. As a result, the position interpolation path is on a planecontaining three points, A, B and O. According to the B-splinedefinition, the viewing-line-of-sight generation/updating unit 5 uses A,B and P to calculate the interpolation updating point Q on the midpointon the segment connecting the midpoints of the segment AP and segmentBP. Then, the viewing-line-of-sight generation/updating unit 5recursively calculates interpolation updating points in the followingtwo directions as interpolation curves: a set in which two endpoints areA and Q and the midpoint of the segment AP is a calculation controlpoint, and a set in which two endpoints are B and Q and the midpoint ofthe segment BP is a calculation control point. A B-spline curve iscalculated as the above process is recursively repeated.

Incidentally, as illustrated in FIG. 5(a), when the control referencepoint that is based on the driver's vehicle is used for calculating thereference point (calculation reference point) of the interpolation pathof the line-of-sight position, it becomes possible to take into accounta maximum distance between the curve, which turns out to be theinterpolation path, and the driver's vehicle. That is, it becomespossible to specify an interpolation line-of-sight path that turns outto be a line-of-sight position that is most remote from the driver'svehicle. Therefore, it is possible to control the change in the apparentsize of the driver's vehicle that occurs due to the change in thepositional relationship of the driver's vehicle viewed from theline-of-sight position and line-of-sight position.

The calculation control point P may be calculated by other methodsincluding those illustrated in FIGS. 5(b) and 5(c) instead of the methodillustrated in FIG. 5(a). Incidentally, in the case of FIG. 5(b), thecontrol reference point O is not used; a separately-calculated pointthat is at an arbitrary distance on a line extending the line-of-sightdirection vector at the midpoint M on the segment AB is regarded as thecalculation control point P. In the case of FIG. 5(c), a point that isat an arbitrary distance on a line that passes through the midpoint Mand is perpendicular to the segment AB is regarded as the calculationcontrol point P. According to the method illustrated in FIG. 5(b), theposition interpolation path exists on a plane defined by theline-of-sight vectors at points A and B and midpoint M. According to themethod illustrated in FIG. 5(c), the position interpolation path existson a plane defined by vectors calculated from the points A and B and theline-of-sight vectors at points A and B (a mean vector of two vectorsand the like, for example).

In this manner, the viewing-line-of-sight direction vector is used forcalculating the control point (calculation control point) of theinterpolation path of the line-of-sight position. Therefore, it ispossible to specify, when the interpolation path is interpolation (curveinterpolation) of second or greater order, the direction of deviation ofthe interpolation line-of-sight position from the linear interpolationin which the path is straight as the distance of the interpolation lineof sight with respect to the viewing-line-of-sight direction. That is,in order to specify the distance in the current viewing direction, itbecomes easier to grasp the change in the viewing size of a subject thatoccurs due to the change of the interpolation position, and it becomeseasier to imagine the interpolation results.

In FIG. 5, what is illustrated as an example of interpolation of aplurality of orders is the B-spline interpolation in which the number ofcontrol points is small. However, such interpolation methods as othergroups of curves like Bezier curves and typical N-order interpolation(spline interpolation, Lagrange interpolation, Newton's interpolationand the like) may be applied as long as the interpolation methods passthrough the visual point positions A and B. As a group of control pointspreferable for N-order interpolation, instead of the midpoint Millustrated in FIG. 5 and the calculation control point P on the linepassing through the midpoint M, a group of control points calculatedfrom a plurality of points at arbitrary positions on the segment AB maybe used. For example, if two control points are preferable, two pointsA′ and B′ that are away from both ends of the segment AB by an amountequivalent to 15 percent of the length of the segment AB may be usedinstead of the midpoint M, and two calculation control points may becalculated from a group of lines passing through the points.

FIG. 6 illustrates an example in which an arc of an ellipse is used asthe position interpolation path. In a similar way to that in FIG. 5(c),the viewing-line-of-sight generation/updating unit 5 calculates a planewhere the arc of the ellipse exists from the visual point positions Aand B and a vector calculated from the line-of-sight vectors of thevisual point positions A and B. The viewing-line-of-sightgeneration/updating unit 5 determines how the arc of the ellipse isdisposed with respect to the segment AB connecting the visual pointpositions with the use of a line that passes through the midpoint M ofthe segment AB and is perpendicular to the segment AB. That is, theviewing-line-of-sight generation/updating unit 5 sets the ellipse'scenter C at an arbitrary position on the line that passes through themidpoint M of the segment AB connecting the visual point positions ofthe viewing line of sight and is perpendicular to the segment AB;calculates the longer and shorter diameters of the ellipse that passesthrough the visual point positions A and B; and regards the ellipse'sarc AB as the position interpolation path. Since the ellipse's arc,which is a part of the ellipse, is regarded as the positioninterpolation path, it is easier to calculate the interpolation updatingposition. Instead of the ellipse's arc, the following method may be usedto calculate the interpolation path: an interpolation path (an arbitrarytrajectory made up of a curve on a plane like the ellipse's arc or aline) for a given segment is determined in advance; the interpolationpath is extended according to the actual segment AB; and a plane onwhich the interpolation path exists is rotated and disposed on athree-dimensional space with the use of the control reference point,midpoint, perpendicular line and viewing direction vector in the sameway as that in the example of FIG. 5.

The viewing-line-of-sight generation/updating unit 5 calculatesparameters that are in line with the interpolation path calculated inthe above way.

In the example illustrated in FIGS. 4 to 6, the interpolation path isdepicted as a curve that could be as differentiable as possible so as tomatch the smooth change of the line of sight. However, the interpolationpath is not limited to the above. The viewing-line-of-sightgeneration/updating unit 5 may calculate a zigzag line that makes apolygon serve as the interpolation path; or a complex path having acombination of one or more lines and one or more curves. In this case,an interpolation path that needs to pass through a specific line ofsight, such as the above pseudo driver's line of sight, along the waymay be calculated.

FIGS. 4 to 6 illustrate an example of the interpolation process based onthe visual point positions. However, a similar interpolation calculationprocess may be applied to any parameters other than the visual pointpositions (viewing direction, angle of field, focal distance and thelike). The calculation method of the interpolation contents may bedifferent except for whether the interpolation is performed for eachparameter: For example, when the interpolation of two parameters, thevisual point position and the line-of-sight direction, is performed, theexample of FIG. 5(a) is used as the interpolation method of the visualpoint position, while the example of FIG. 4 is used as the interpolationmethod of the line-of-sight direction.

The interpolation of a parameter having one value, such as the focaldistance, may be performed in an arbitrary way. However, in a similarway to the speed of the interpolation description of the visual pointpositions, the value may be uniformly changed during the interpolationprocess. The change rate may vary so that the rate of change is slowerat the start and end of the interpolation process than in the middle ofthe interpolation process.

Incidentally, according to the flowchart, the viewing-line-of-sightgeneration/updating unit 5 calculates the contents of how to perform theinterpolation and updating of the viewing line of sight every time.However, if the viewing lines of sight A and B are linked in advance topredetermined scenes, the interpolation paths of the viewing lines ofsight can also be determined in advance. Accordingly, a part or all ofthe interpolation calculation of the parameters of the viewing lines ofsight may be performed in advance, and the updating of the viewing linesof sight may be performed by referring to the values.

In this manner, the viewing-line-of-sight generation/updating unit 5calculates the interpolation path of the viewing line of sight andupdates the viewing-line-of-sight parameters when preferable so that theviewing-line-of-sight parameters are in line with the path. Therefore,it is possible for the vehicle image processing device 100 to realize asmooth movement of the visual point. For example, as illustrated in FIG.7, the vehicle image processing device 100 generates a midair overheadimage of the driver's vehicle with the viewing line of sight A to mainlyview the road surface around the driver's vehicle; generates a vehicleforward image with the viewing line of sight B to mainly view not onlythe road surface but the general surrounding situation in the forwarddirection; generates the line-of-sight movement paths corresponding tothe viewing lines of sight A and B whose viewing areas and directionsare different; and updates the viewing-line-of-sight parameters whenpreferable so that the viewing-line-of-sight parameters are in line withthe path. Therefore, it is possible to generate a changed image in whichthe area and direction of the viewed image smoothly change as the visualpoint smoothly moves.

Returning to the explanation of the flowchart of FIG. 2, after thechanging and updating of the line of sight, the drawing methoddetermination unit 6 makes a determination as to whether to change thedrawing method of the captured image that is a target to be drawn andchanges the actual drawing method if the method is to be changed (StepsS13, S14 and S15). Here, for the purpose of convenience, a method ofconverting some image or shape to another image to be viewed is referredto as a drawing method; a target that is to be converted by the drawingmethod is referred to as a drawing target. There are two types ofdrawing method that the drawing method determination unit 6 candetermine. One method is to calculate in advance the relationship(referred to as a projection map in this case) between the position ofeach pixel inside the captured image that is a drawing target and thepixel position inside the drawn image to which the pixel is linked; andallocate the actual pixel color information of the drawing-target imagein accordance with the calculated relationship as if a simple task isdone. The other method is a similar drawing method to a texture mappingof a typical computer graphics (referred to as CG, hereinafter) forshapes and uses one or more three-dimensional shapes as projectionshapes to which the captured image is attached in order to realize amore complex relationship. For the purpose of convenience, the formerdrawing method is referred to as a drawing method that uses a projectionmap; the latter drawing method is referred to as a drawing method thatuses shape.

As a way of determining a drawing method, the drawing methoddetermination unit 6 makes a determination as to whether there is datafor using the drawing method that uses the projection map, i.e. theprojection map exists for the viewing line of sight (Step S13). When thedata exists (Step S13, YES), the drawing method determination unit 6uses the drawing method that uses the projection map (Step S14). Whenthe projection map does not exist (Step S13, NO), the drawing methoddetermination unit 6 uses the drawing method that uses projection shape(Step S15).

If the viewing lines of sight are different, the projection map isanother data. In order to support various viewing lines of sight, it isdesirable that many projection maps be retained. However, in reality, itis impossible given the storage capacity. Accordingly, suppose that thevehicle image processing device 100 of the present embodiment holdsprojection maps that are calculated in advance for a group of viewinglines of sight corresponding to each usage scene as described in thesection of the viewing-line-of-sight changing unit 4. A frequently-usedscene is predicted in advance and a projection map is prepared for thecorresponding group of viewing lines of sight. Therefore, it is possibleto use the efficient drawing method that uses a projection map so thatthe amount of data, as well as the amount of calculation, is small.

Incidentally, the changing of the viewing line of sight detected by theviewing-line-of-sight changing unit 4 is mainly the changing of the lineof sight in any scene other than the one in which the user gives adirect instruction as described above. Therefore, it is highly likelythat the unchanged first image made up of the viewing line of sight andthe changed second image have a projection map, and that the drawingmethod that uses the projection map is determined as the drawing method.Meanwhile, in many cases, the changed image (interpolation image)created with the use of the line-of-sight parameter generated by theviewing-line-of-sight generation/updating unit 5 is made of the viewingline of sight generated after the interpolation is performed betweenscenes or of the viewing line of sight to which fine adjustments aremade and does not have a projection map. Therefore, in many cases, thedrawing method that uses shape is determined as the drawing method butis less frequently used. Incidentally, the details of the projection mapdata, the usage of the data, and the contents of the projection shapewill be described later.

By the way, if the drawing target is a captured image of a camera otherthan the vehicle-mounted camera mounted on the driver's vehicle, it isdifficult to use the drawing method that uses a projection map. Thereason is that the projection map turns out to be another data as thepositional relationship between the camera that takes a picture and theviewing line of sight changes. Even if the viewing line of sight doesnot change as a line of sight that looks down from right above to rightbelow the driver's vehicle, the position of the captured image occupyingthe drawn viewing image changes as the positional relationship betweenthe image-capturing camera and the driver's vehicle changes because ofthe movement of the driver's vehicle. Therefore, the projection map,which represents the positional relationship between the pixel positionof the captured image and the pixel position of the drawn image, turnsout to be another data. Therefore, as illustrated in FIG. 17, when theviewing line of sight is set relative to the driver's vehicle, i.e.,when the viewing line of sight automatically moves as the driver'svehicle moves, the driver's vehicle is basically considered to move.Therefore, regardless of whether the viewing line of sight is changed ornot, the drawing method that uses shape is used when the drawing targetis a captured image of a camera other than the vehicle-mounted camera.

The related object determination unit 7 determines a related object tobe used and a drawing method thereof (from step S16 to step S22). Therelated object is used in viewing the state of the driver's vehicle, thesituation around the driver's vehicle and information, such as thetraveling direction of the driver's vehicle, the position and size ofthe obstacle as well as the direction in which the obstacle is locatedrelative to the driver's vehicle, the direction of the destination ofthe driver's vehicle, the position of the destination, traffic lightsaround the driver's vehicle, traffic signs such as signs and pavementsigns, attention areas such as an intersection where the vehicle isscheduled to turn left or right, congestion/accident-prone areas,recommended shops and other kinds of feature and map information. Theinformation can be acquired from the driving information acquisitionunit 1, the road information acquisition unit 2, the directspecification unit 3 or the like. There are two object: an object forrepresenting the existence and contents of information, and an objectfor representing the information and contents preset by the systemmainly as predetermined information, such as the driver's vehicle, thecamera mounted on the driver's vehicle and the blind spot of the camera,which are not acquired from the driving information acquisition unit 1,the road information acquisition unit 2, and the direct specificationunit 3.

Incidentally, if a captured image of a camera other than the driver'svehicle's is used, a camera that acquires a camera parameter such asposition as well as the captured image at once and the camera's blindspot may be regarded as a related object along with the above two.

The related object determination unit 7 confirms whether there is theone that matches a to-be-displayed related object that has been definedin advance, by referring to the input vehicle's state (vehicleinformation) and doing other processes (Step S16). If the one matchesthe to-be-displayed related object (Step S16, YES), the one is added toa usage list and it is determined that the one is to be used (Step S17).

The related object determination unit 7 for example confirms that thereis to-be-displayed data of the related object of the obstacle for theabove position and size of the obstacle input from sensors and the likeand the direction in which the obstacle is located relative to thedriver's vehicle. Furthermore, based on a preset usage rule such as thefollowing one and the like, the related object determination unit 7determines whether to use the related object: if the related object isfor example located within a traveling predicted lane of the driver'svehicle or for example within 5 m to the left or right of the centerline of a traveling predicted lane of the map data whose prediction isbased on the position of the driver's vehicle and the travelingdirection and the size of the related object is grater than or equal to10 cm, the related object is to be displayed. Instead of using such acomplex determination rule, all related objects may be displayed ifthere is to-be-displayed data of the related objects.

The related object is made up of the state and contents of the driver'svehicle as well as the state and contents around the driver's vehicle,position data representing the positional relationship about where therelated object is located around driver's vehicle, and to-be-displayeddata used for drawing. The to-be-displayed data of the related objectconsists of at least one or more of the following components: a preparedimage and a three-dimensional shape. Even when only a character string,which is the contents of information, is to be displayed, font images ofcharacters that make up the character string are regarded as images ofthe related object for the purpose of convenience. Incidentally, forexample, since the driver's vehicle is stuck in traffic congestion,there is information that is exceptionally unclear about the positionalrelationship from the driver's vehicle and does not have a clearposition, such as an estimated time that the driver's vehicle would taketo get out of the traffic jam and other kinds of information. However,the vehicle image processing device 100 may supplement, when preferable,the position to be an arbitrary position, which is for example aposition specified by the system, before using the position. If thethree-dimensional shape is used as display data, information fordetermining the color of the shape (which for example includes the colorof each vertex and a vertex normal line, as well as texture mapping dataif preferable) may be retained at the same time along with the shapethat is the coordinate value of a vertex inside the three-dimensionalspace.

According to the present embodiment, in a similar way to the aboveprojection map of the captured image, the following is retained as thedisplay data of the related object: an image of the related object for agroup of viewing lines of sight corresponding to each preset usagescene.

Incidentally, the usage list of the related object is reset each time aprocess comes in the related object determination unit 7, i.e. thedrawing process is performed, because the related object is regarded aszero. However, the contents of the usage list used in the previousdrawing process may be appropriately changed for reuse. When the usagelist is reused, it is preferable to make sure that the retained data,particularly information, of the related object reflects the change ofthe position data.

According to the present embodiment, in order to make the processsimple, the process of determining the drawing method of the nextrelated object is performed after all the to-be-used related objects areexamined and added to the usage list. However, the drawing method may bedetermined each time one to-be-used related object is found. In thiscase, the usage list could be not preferable if the configurationenables the process to be performed for each related object.

Then, the related object determination unit 7 sequentially checkswhether an unprocessed related object is in the usage list (Step S18).When there is an unprocessed related object (Step S18, YES), the relatedobject determination unit 7 determines a drawing method by referring tothe to-be-displayed data of the related object (from step S19 to stepS22). In this case, the related object determination unit 7 makes adetermination as to whether there is the shape of the related object(Step S19). When there is no shape and only an image exists (Step S19,NO), the related object determination unit 7 selects the drawing methodthat uses an image (Step S22). When there is a shape (Step S19, YES),the related object determination unit 7 makes a determination as towhether to prioritize and select the drawing method that uses shape(Step S20). When the related object determination unit 7 prioritizes andselects the drawing method that uses shape (Step S20, YES), the relatedobject determination unit 7 selects the drawing method that uses shape(Step S21). Incidentally, when the related object determination unit 7does not prioritize and select the drawing method that uses shape (StepS20, NO), the related object determination unit 7 selects the drawingmethod that uses an image (Step S22).

FIG. 8 illustrates, as related objects, an example of the image andshape that are display data for representing the position of thedriver's vehicle. FIG. 8(a) depicts the three-dimensional shape of thedriver's vehicle, which consists of triangular polygons in the example.FIG. 8(b) illustrates images of the driver's vehicle: according to thepresent embodiment, an image that is seen from above, an image that isseen in the lateral direction, and two images that are diagonally seenare prepared so as to correspond to the viewing lines of sight A to Dillustrated in FIG. 17. Incidentally, the number of images is notpreferably the same as the number of scenes. The number of images may bean arbitrary value that is equal to or greater than one.

FIG. 9 illustrates related objects indicating the existence ofobstacles, as an example of the objects for indicating the existence andcontents of information acquired from the driving informationacquisition unit 1, the road information acquisition unit 2, the directspecification unit 3 and the like. As the related objects indicated byarrows or hands, FIGS. 9(a) and 9(b) depict the three-dimensionalshapes, while FIGS. 9(c) and 9(d) depict three types of the image.

The process of determining whether to prioritize and select the drawingmethod that uses shape is to make a determination as to which drawingmethod is prioritized and used when both image and shape are retained asdata of the related object as illustrated in FIGS. 8 and 9. As for thedrawing method that uses shape, the increased cost of drawing is takeninto account when the determination is made in large part because theamount of calculation increases. For example, when the determinationresult of the drawing method of the captured image by the drawing methoddetermination unit 6 leads to the drawing method that uses a projectionmap, the related object determination unit 7 prioritizes the drawingmethod that uses an image over shape as the drawing method of therelated object. The reason is that the prioritization and determinationprocess is to make sure the following consideration is reflected: thecost of drawing with the use of shape is expensive and, when the shapeis not used in the drawing of the captured image, the shape is not usedin the drawing of the related object. If the determination result by thedrawing method determination unit 6 leads to the drawing method thatuses a projection shape, the related object determination unit 7prioritizes shape as the drawing method of the related object.

As a prioritization/selection determination process other than theabove, the related object determination unit 7 may regard the cost ofthose involving many polygons of the shape of the related object asexpensive and use an image. The related object determination unit 7 mayexamine the current processing load (CPU, memory or data-transferringload) and may use an image when detecting that the load is high.Moreover, the following is also possible: depending on whether theinformation is of a type that stands out or needs to be emphasized at atime when the information is viewed, such as the driver's vehicle thatis frequently enlarged when being displayed or the obstacle that needsto be emphasized, it is determined that the shape is prioritized ifthere is the shape. The vehicle image processing device 100 determinesthe drawing method of the related object after going through such aprioritization/selection determination process. However, theprioritization/selection determination process may be omitted.

Incidentally, the drawing method determination unit 6 regards thecaptured image as a drawing target and determines the drawing method;the related object determination unit 7 regards the related object as adrawing target and determines the drawing method. However, the above wayof separation and the processing order are one example. Both may beperformed at the same time. The use of the related object and theprocess of determining the drawing method may be performed in advance.

Returning to the explanation of the flowchart of FIG. 2, the drawingunit 20 performs a drawing process and the display unit 16 performs adisplay process based on the drawing methods of the captured image andrelated object determined by the drawing method determination unit 6 andthe related object determination unit 7 (Step S23) before the processends. The drawing process, the display process and the subsequentprocesses will be described in detail with reference to FIG. 3.

Incidentally, according to the flowchart illustrated in FIG. 2, thedrawing is performed even when there is no change in drivinginformation, road information and directly-specified contents (Step S5,NO), i.e. when the surrounding environment of the driver's vehicle andthe position of the driver's vehicle do not change, the user does notspecify and the viewing visual point does not change (Step S7, NO).However, the process may return to START and not perform the drawing.However, in this case, attention needs to be paid to the following:subtle changes, such as the changes of the external environment of thedriver's vehicle that may not probably detected from driving informationor road information, which include the swaying of leaves of roadsidetrees in the wind and the change of the color of a traffic signal, donot appear on a final image and thus may not be viewed when the drawingis not performed, even if the subtle changes have been captured by acamera and retained as a captured image.

With reference to the flowchart of FIG. 3, the drawing process of thedrawing unit 20 and the display process of the display unit 16 (theprocess of step S23 of FIG. 2) will be described.

Based on the drawing methods of the captured image and related objectsdetermined by the drawing method determination unit 6 and the relatedobject determination unit 7, the drawing unit 20 performs the drawing onthe basis of the generated line-of-sight parameter when the process hasbeen performed by the viewing-line-of-sight generation/updating unit 5(from step S117 to step S125). A determination is made as to whether aprojection shape is used in the drawing of the captured image (StepS117). If the projection shape is used (Step S117, YES), the projectionshape setting unit 10 disposes the projection shape (Step S118). Whenthe projection shape is not used (Step S117, NO), the projection imageconversion unit 11 performs the conversion of the captured image usingthe projection map (Step S119).

Here, the processes of the projection image conversion unit 11 andprojection shape setting unit 10 will be described. With reference toFIGS. 10, 11 and 12, the following describes the conversion of thecaptured image with the use of the projection map by the projectionimage conversion unit 11.

FIG. 10(a) illustrates an overhead image seen from the sky above thedriver's vehicle, as an example of a to-be-viewed projection imagecreated by the projection map's drawing. Incidentally, the upwarddirection of the image represents the traveling direction of thedriver's vehicle. FIG. 10(b) is a schematic diagram of FIG. 10(a) andillustrates, like FIG. 10(a), where a group of captured images of FIG.10(c) is projected on the projection image. FIG. 10(c) depicts groups ofcaptured images A to D, captured by front, rear, left and right camerasA to D, that are drawing targets. The projection image conversion unit11 for example generates from each of the captured images illustrated inFIG. 10(c) a projection image as illustrated in FIG. 10(a), whichmatches the current viewing line of sight, or the viewing line of sightthat overlooks from the sky above the driver's vehicle in this case.

Here, in the captured images of FIG. 10(c), a pixel P (X1, Y1) of theprojected image of FIG. 10(a) is liked to two, A1 (A1 x, A1 y) insidethe captured image A and D1 (D1 x, D1 y) inside the captured image D;Q(X2, Y2) is linked to two, A2 (A2 x, A2 y) inside the captured image Aand C1 (C1 x, C1 y) inside the captured image C. Incidentally, as oneexample, both P and Q are linked to a plurality of captured images.However, P and Q may be linked to one captured image or three or morecaptured images.

With reference to FIGS. 11 and 12, an example of a projection maprepresenting the relationship between pixels inside the imageillustrated in FIG. 10 will be described. FIG. 11 depicts an example inwhich even the projection image is generated for each captured imagewith the use of the projection map for each captured image as aprojection map. According to the present embodiment, each projection mapis made up of the positions of pixels inside the projection image, thepositions of pixels inside the captured image and weighting factors thatare used in calculating the color of pixels. The weighting factorrepresents the priority of each pixel to determine what color is usedwhen the pixels of a plurality of captured images are linked to the samepixel position inside the projection image. However, the weightingfactor may be omitted on the assumption that everything has the samepriority. In the example illustrated in FIG. 11, the weighting factor isapplied to the degree of transparency without change (from 0.0 to 1.0;1.0 means “obscure,” and 0.0 “transparent”).

The projection image conversion unit 11 generates a projection image Ausing a captured image A captured by the camera A and a correspondingprojection map A as well as a projection image B using a captured imageB captured by the camera B and a corresponding projection map B. Theprojection image conversion unit 11 similarly generates a projectionimage C and a projection image D. By combining the above images, theprojection image conversion unit 11 generates a projection image(combined projection image). Incidentally, in the projection imagegenerated by the projection image conversion unit 11, a portion that hasno corresponding pixels inside the captured image is regarded as atransparent area so that the portion is not taken into account.

Incidentally, in the example where the projection map is described, onepixel is linked to one pixel. However, a pixel area consisting of aplurality of pixels may be liked to a pixel area consisting of aplurality of pixels. Or alternatively, one pixel may be liked to a pixelarea. The projection map is not limited to a table illustrated in FIG. 1as long as it is possible to recognize the relationship between pixels.If there is extra space in the storage capacity, a pixel position dataarray whose size is the same as the resolution of one of the images maybe secured and, only for a portion where there is a pixel of acorresponding image, the position of the pixel of the correspondingimage may be written down at the position inside the data array thatcorresponds to the position of the above pixel. Moreover, the verticallength and lateral length of the image may be normalized and regarded as1.0; the position inside the image that does not preferably correspondto a strict pixel, like the pixel that is positioned 0.5 from the topand 0.25 from the left, may be specified. In this case, as the color ofthe pixel corresponding to the specified position, the color of thesurrounding pixels may be acquired by an arbitrary existinginterpolation method such as a bilinear method.

FIG. 12 illustrates an example in which one projection image isgenerated from a plurality of captured images with the use of aprojection map that is used for all captured images as a projection map.Like the one illustrated in FIG. 11, the projection map for all camerasconsists of the positions of pixels inside the projection image, theimage IDs of the corresponding captured images, the positions of pixelsinside the actual captured image and weighting factors. Therelationships inside the projection map are arranged in arbitrary order.The weighting factors carry the same meaning as what is described withreference to FIG. 11; the weighting factors may be similarly omitted.The projection image conversion unit 11 refers to the projection map;merges the colors of pixels while weighting each pixel inside eachcaptured image with a weighting factor; and generates a projectionimage. For example, as illustrated in FIG. 12, the color of the pixel ofC1 is multiplied by a weighting factor of 1.0, the color of the pixel ofA2 is multiplied by a weighting factor of 0.8, and then the color of thepixel at point Q is obtained by normalizing the above colors of pixels.In the case of FIG. 12, the linking may be performed with the use ofpixel areas as in the case of FIG. 11 and is not limited to what isdescribed on the table.

If a projection area (an area on a captured image that is used in aprojection image) corresponding to each line-of-sight parameter and aprojection map are set in advance for an interpolation image between theunchanged and changed images (first and second images), it is possiblefor the projection image conversion unit 11 to carry out the aboveprocess.

The following describes a method used by the projection shape settingunit 10 to dispose a projection shape.

The projection shape setting unit 10 sets an arbitrary projection shapein a three-dimensional space relative to the driver's vehicle; and usesa camera parameter of each captured image to set which pixel of whatcaptured image is linked to which position of the projection shape inthe same way as an existing texture mapping of computer graphicsspecifies an attachment position. More specifically, the positionalrelationship between an image-capturing plane of a camera and theprojection shape is calculated from a given projection shape (apredetermined projection shape) disposed relative to the driver'svehicle and camera parameters such as the position of the camerarelative to the driver's vehicle and the direction of the camera.Projection calculation is performed to figure out where the camera'simage-capturing plane (=the positions of pixels inside the capturedimage) are to be projected on the projection shape with the use of atypical projection method such as perspective projection in order tocalculate how the positions of pixels are linked.

For example, take a line connecting the position of the camera and apoint on the image-capturing plane that corresponds to each pixel of thecaptured image, and calculate a point where the line and the projectionshape cross each other. The coordinate of the intersection on theprojection shape is liked to the position of a pixel inside the capturedimage that corresponds to a point on the image-capturing plane. In theexample here, the coordinate of the projection shape is calculated fromthe point on the image-capturing plane (the position of each pixelinside the captured image). However, the following is also possible: aline connecting an arbitrary characteristic point of the projectionshape to the position of the camera is calculated and, if the linecrosses the image-capturing plane of the camera, the position of a pixelof the captured image that corresponds to a point of the image-capturingplane that the line crosses is liked. In this case, the position of thepixel is the position inside the image that does not preferablycorrespond to one precise pixel. Moreover, the color of the pixel at theabove position of the pixel can be acquired in the same way as what isdescribed with reference to FIG. 11.

FIG. 13 illustrates an example of a projection shape consisting of aspheroidal surface and a plane representing a road surface. A vehicle onthe right side of FIG. 13 is the driver's vehicle illustrated forreference to indicate the positional relationship between the vehicleand the projection shape, not a projection shape. In the example here,the projection shape is made up of the spheroidal surface, which is asurrounding three-dimensional portion onto which a surrounding image ismainly projected, and the plane, which is a plane overlooked portion onwhich the road around the driver's vehicle is mainly projected. However,the projection shape is not limited to this. The projection shape may bean arbitrary plane, an arbitrary curved surface, or a combination of theplane and the surface. The positional relationship between theprojection shape and the driver's vehicle may be arbitrarily set.

In this manner, the projection shape setting unit 10 is different fromthe projection image conversion unit 11: What is calculated by theprojection shape setting unit 10 is the liking of the position of pixelsthat can be calculated only from the relationship between theimage-capturing camera and the projection shape regardless of thecurrent viewing line of sight. The linking remains unchanged unless theprojection shape changes or the positional relationship between theimage-capturing camera and the projection shape changes. Once thelinking is calculated, the calculation results may be repeatedly usedwithout performing the calculation again. However, in the case where theprojection shape is set relative to the driver's vehicle as illustratedin FIG. 13, i.e. the case where the projection shape automatically movesas the driver's vehicle moves, if a captured image of a camera otherthan the camera mounted on the driver's vehicle is used, the positionalrelationship between the image-capturing camera and the projection shapechanges as the driver's vehicle moves. Therefore, it is preferable toperform the calculation every time. The liking of the positions ofpixels calculated by the projection shape setting unit 10 is used by theprojection conversion unit 14 described below, where an image seen alongthe viewing line of sight is created with the use of information of theviewing line of sight.

Returning to the explanation of FIG. 3, the drawing process of therelated object is then performed. The process continues until there isno related object to be displayed. When there is an unprocessed relatedobject (Step S120, YES) and when the drawing method that uses shape isselected at step S21 for the related object (Step S121, YES), therelated object shape setting unit 12 sets the shape of the relatedobject in the three-dimensional space relative to the same driver'svehicle as that of the projection shape with the use of positionalrelationship data of the related object and driver's vehicle (StepS122), and returns to the process of making a determination as towhether there is an unprocessed related object (120 of the flowchart).

The process of the related object shape setting unit 12 will bedescribed with reference to FIG. 14. Incidentally, FIG. 14(a) is aperspective view of a model space that is a to-be-processed area; FIG.14(b) is a bird's-eye view of the model space.

The following shapes are disposed in the same three-dimensional space:the projection shape consisting of a plane and a spheroidal surface thatis centered at the center of the driver's vehicle (the reference pointof the driver's vehicle) as illustrated as an example in FIG. 13, theshape of the driver's vehicle, and the shapes of two related objectsthat are in the shape of an arrow indicating obstacles. For example, therelated object shape setting unit 12 uses the actual position of anobstacle obtained from a sensor or the like and disposes an arrow shapeas a related object shape around the position of the obstacle in thedirection of a vector from the driver's vehicle to the obstacle(existence-direction vector). At this time, if the shape of the relatedobject is disposed outside the projection shape when seen from thedriver's vehicle, the related object could not been seen because therelated object may be positioned behind the captured image projectedonto the projection shape. Therefore fine adjustments may be made sothat the related object is disposed inside the projection shape.

Returning to the explanation of FIG. 3, when the drawing method thatuses an image (Step S22) is used for the related object instead of thedrawing method that uses shape (Step S21, NO), the related object imagesetting unit 13 selects the image of the related object in accordancewith the current viewing line of sight; generates a related objectdisplay image where the image is disposed based on the positionalrelationship between the viewing line of sight and the related object(Step S123); and returns to the process of making a determination as towhether there is an unprocessed related object (step S120).

FIGS. 15 and 16 illustrate one example of disposing the related objectimage as well as generating the display image. FIG. 15(a) is a schematicdiagram illustrating the driver's vehicle seen from the right direction,the viewing line of sight (referred to as the viewing line of sight A)from the visual point position A, and an obstacle that is an example ofthe related object (illustrated here as a quadrilateral). FIG. 15(b) isa schematic diagram illustrating the same things as those in FIG. 15(a)but seen from the sky above the driver's vehicle. The viewing line ofsight A overlooks the driver's vehicle diagonally from the top left.Suppose the obstacle corresponding to the related object is positionedat P and that a vector extending from the reference point O of thedriver's vehicle to the obstacle P is regarded as an existence-directionvector.

FIG. 16(a) depicts the related object images used to display theobstacle. In the example here, an image A is an image viewed fromdirectly above, an image B is an image viewed diagonally from behind,and an image C is an image viewed in a direction that is closer to thebackward direction than the image B is viewed. The image data are notjust pictures. An image attachment origin D and an image attachmentdirection vector E are set in advance: an image is to be attached basedon the image attachment origin D and the image attachment directionvector E. Incidentally, the image attachment origin D of the image C ispositioned slightly above the bottom. In this manner, the positions ofthe image attachment origins D of the images A, B and C are not allpositioned at the centers of the bottom lines of the images, and thepositional relationships of the image attachment origins D and imageattachment direction vectors E are not the same. However, the positionalrelationships may be the same.

FIG. 16(b) is an example of a table used in making a determination as towhich group of images illustrated in FIG. 16(a) is used.

For example, in FIG. 15, on a plane containing the position P of theobstacle, the visual point position A and the reference point O, supposethat an angle formed by a vector extending from the visual pointposition A to the obstacle P and a vector extending from the referencepoint O to the obstacle P, or existence-direction vector, is α: α is anapparent angle of the obstacle viewed from the line-of-sight position A.FIG. 16(b) illustrates a table indicating which image is to be used fromthree types of images illustrated in FIG. 16(a), depending on the valueof the apparent angle α. If the value of α is greater than or equal to60 degrees and less than 120 degrees, the related object image settingunit 13 uses the image A that is an image viewed from directly above.

Incidentally, according to the present embodiment, the determination ismade by the related object image setting unit 13. However, the unit thatmakes the determination is not limited to the related object imagesetting unit 13. The determination may be made by the above relatedobject determination unit 7.

FIG. 16(c) illustrates an example of the related object display imagemade by actually disposing the selected image in accordance with theposition where the related object exists. As illustrated in the leftportion of FIG. 16(c), the related object image setting unit 13 firstcalculates, using the line-of-sight parameter of the viewing line ofsight A, where the reference point O, the obstacle P and theexistence-direction vector are to be projected in the projection imagethat uses the viewing line of sight A. In this case, in the projectionimage viewed along the viewing line of sight A that diagonally overlooksfrom the left rear, the existence-direction vector extends from thedriver's vehicle to the top left. The related object image setting unit13 uses the calculated position of the obstacle P and theexistence-direction vector to place the image attachment origin D ofFIG. 16(a) at the position P of the obstacle; disposes the imageattachment direction vector E in the same direction as theexistence-direction vector of the related object; and generates an imageillustrated in the right portion of FIG. 16(c) that is a related objectdisplay image. Incidentally, the related object image setting unit 13makes the color of a portion where the related object image is notdisposed transparent in the related object display image as illustratedin the right portion of FIG. 16(c).

Incidentally, in the example here, for simplicity of explanation, therelated object display image having a large transparent portion iscreated in the same size as the projection image for each relatedobject. However, the related object display image and the projectionimage are not preferably the same in size. In order to reduce thestorage capacity, the following is possible: the related object displayimage is made as a smaller image with a reduced transparent portion, theposition where the related object display image is to be disposedrelative to the projection image and the like are determined, and therelated object display image is superimposed by the image superimposingunit 15 described later by referring to the disposition position and thelike. The related object display image may not be generated. The colorof the related object image may be directly written to the followingimage by the image superimposing unit 15 described later by referring tothe disposition of the related object determined: the final viewingimage or an arbitrary image that is used as a material to make the finalviewing image through superimposing, blending and other processes, suchas another related object display image or a projection imagecorresponding to a captured image of a given camera.

After the processes are performed by the related object shape settingunit 12 and the related object image setting unit 13, whether there isan unprocessed related object is checked again. If there is nounprocessed related object (Step S120, NO), the projection conversionunit 14 performs the following process when the projection shape is usedin the drawing method of the captured image (Step S117, YES) or/and whenthe shape is used in the drawing method of the related object (StepS121, YES): the process of making a projection conversion image reflectthe actual color of pixels of the captured image with the use of therelationship with the captured image preset in the projection shape,converting the captured image and the related object shape that is setat the same time into an image seen along the current viewing line ofsight with the use of an already known projection method such asperspective projection, and generating the projection conversion image(Step S124).

For example, the projection conversion unit 14 makes a determination asto whether a line passing through the visual point of the currentviewing line of sight and a point inside a pseudo image-capturing planeof the viewing line of sight crosses the projection shape and therelated object shape; acquires the color at the intersection of theshape that the line crosses; calculates the color of the pixel positionof the projection conversion image corresponding to the image-capturingplane; and draws the pixel using the color to obtain the projectionconversion image. The calculation method is a calculation method thatuses perspective projection, one example of the already-known projectionmethod, which is also used by the projection shape setting unit 10 incalculating the relationship of the pixel positions. Projectionconversion is therefore realized by the projection conversion unit 14.If the shape that the line crosses is a projection shape, the color atthe intersection can be obtained from the color of the pixel of thecaptured image that is obtained by referring to the relationship withthe present captured image. Incidentally, as the color at theintersection, the color of the related object or the color of theintersection that is the color of the pixel of the correspondingcaptured image may be used without change. Or alternatively, with theuse of a typical CG shading method (drawing color calculation method), amore realistic color may be calculated from the reflectivity of a lightsource or shape that is set for the scene, a surface normal line of theintersection, or the like for use.

If it is found as a result of the intersection determination that theline does not cross the projection shape or related object shape, thecolor of the pixel of the corresponding projection conversion image maybe made transparent or any arbitrary color such as a default backgroundcolor. If the projection conversion unit 14 uses only the related objectshape without using the projection shape, i.e. if a projection imageconversion unit 17 performs the process for the captured image using thedrawing method that uses the projection map while the related objectshape setting unit 12 and the projection conversion unit 14 perform theprocesses for the related object using the drawing method that usesshape, it is especially desirable that the color of the pixel betransparent. Since a portion other than the portion where the relatedobject is projected is made transparent in the projection conversionimage, the color of the lower image can be seen through the portionother than the related object after another image is superimposed by theimage superimposing unit 15 described below. As a result, the projectionconversion image whose drawing target is the captured image generated bythe projection image conversion unit 11 can be superimposed below theprojection conversion image so as to be beautifully visibletherethrough; and an image where the related object appears to bewritten on the converted captured image can be created.

In that manner, the projection conversion unit 14 performs theprojection conversion calculation using the viewing line of sightchanged by the viewing-line-of-sight changing unit 4 and the parameterof the viewing line of sight that is interpolated and updated by theviewing-line-of-sight generation/updating unit 5, thereby making theprojection conversion image of the captured image and related objectviewed along the above viewing lines of sight.

Meanwhile, when the projection shape is not used in the drawing methodof the captured image (Step S117, NO) and when there is no relatedobject or the shape is not used in the drawing method of the relatedobject (Step S121, NO), the projection conversion process by theprojection conversion unit 14 (Step S124) is skipped.

Then, the image superimposing unit 15 superimposes one or moreprojection conversion images generated by the projection imageconversion unit 11 or projection conversion unit 14 as well as a relatedobject display image if there is a related object drawn by the image andoutputs a final output image (Step S125). The display unit 16 displaysthe finally output image generated by the image superimposing unit 15(Step S126). The process of the vehicle image processing device 100 thenends.

As described above, according to the present embodiment, the projectionmap is used in an effective manner for the viewing lines of sight thatchanges according to scenes. Therefore, it is possible to smoothlychange the line of sight with less processing load. For the relatedobject to be displayed at the same time, the shape and image of therelated object are selected for use. Therefore, the displaying can berealized without any sense of discomfort for the smooth changing of theline of sight with less processing load.

Moreover, the vehicle image processing device 100 of the presentembodiment can realize the smooth changing of a viewing line of sightthat may not determined in advance, which is for example the changing toa viewing line of sight that is dynamically calculated, such as a lineof sight that gets closer to an obstacle in such a way that the obstacleis closely observed. At the same time, the vehicle image processingdevice 100 continuously makes a determination as to whether it ispreferable to perform the three-dimensional model calculation bymonitoring a change of the line of sight. Therefore, the power of thecalculation process can be saved.

Incidentally, an image acquisition means corresponds to the group ofcameras 8 and distortion correction unit 9 of the present embodiment. Aprojection conversion means corresponds to the drawing unit 20 (theprojection shape setting unit 10 and the projection conversion unit 14)of the present embodiment. A line-of-sight change detection meanscorresponds to the viewing-line-of-sight changing unit 4 of the presentembodiment. A line-of-sight parameter generation means corresponds tothe viewing-line-of-sight generation/updating unit 5 of the presentembodiment.

A changed image generation means corresponds to the drawing unit 20 (theprojection shape setting unit 10 and the projection conversion unit 14)of the present embodiment. An image conversion means corresponds to thedrawing unit 20 (the projection image conversion unit 11). A relatedobject determination means corresponds to the related objectdetermination unit 7 of the present embodiment. A related object dataselection means corresponds to the related object determination unit 7.

A related object image setting means corresponds to the related objectimage setting unit 13 of the present embodiment. A related object shapesetting means corresponds to the related object shape setting unit 12. Avehicle information acquisition means corresponds to the drivinginformation acquisition unit 1 and road information acquisition unit 2of the present embodiment.

The smooth changing of the virtual line of sight is realized. Thechanged image that gradually changes before and after change isgenerated. Therefore, it is possible for a user to easily grasp thesituation around the vehicle displayed on an image after the virtualline of sight changes.

All examples and conditional language recited herein are intended forpedagogical purposes to aid the reader in understanding the inventionand the concepts contributed by the inventor to furthering the art, andare to be construed as being without limitation to such specificallyrecited examples and conditions, nor does the organization of suchexamples in the specification relate to a showing of the superiority andinferiority of the invention. Although the embodiment(s) of the presentinvention has(have) been described in detail, it should be understoodthat the various changes, substitutions, and alterations could be madehereto without departing from the spirit and scope of the invention.

What is claimed is:
 1. A vehicle image processing apparatus comprising:a memory configured to store a captured image which is an image that hascaptured an area around a vehicle with use of at least one camera; and aprocessor coupled to the memory, the processor configured to: acquirethe captured image; convert, using at least one predetermined projectionshape, the captured image into an image viewed along a virtual line ofsight that is a line of sight running from a predetermined position in apredetermined direction; detect whether a virtual line of sight isswitched from a position of a first virtual line of sight to a positionof a second virtual line of sight which is a different virtual line ofsight, the position of the first virtual line of sight associated with aparameter value of at least one selected from a predetermined positionand a direction of the virtual line of sight and the position of thesecond virtual line of sight being associated with a parameter valuedifferent from the parameter value associated with the position of thefirst virtual line of sight; acquire at least one type of parametervalue associated with a virtual line of sight for each position of thefirst virtual line of sight and the second virtual line of sight afterdetecting that the virtual line of sight is switched from the positionof the first virtual line of sight to the position of the second virtualline of sight, and generate a parameter value of a position of a virtualline of sight that is gradually changed along a predetermined pathconnecting the position of the first virtual line of sight and theposition of the second virtual line of sight; convert the captured imageto an image viewed along at least one virtual line of sight among thefirst virtual line of sight and the second virtual line of sight using aprojection map which includes a relationship between a position of apixel of the captured image and a position of a pixel of an image viewedalong the first virtual line of sight or the second virtual line ofsight, and generate the converted image to be viewed from the virtualline of sight included in the virtual line of sight group changing fromthe first virtual line of sight to the second virtual line of sight notusing a projection map but using a predetermined projection shape withthe linking of the position of pixels calculated by the positionalrelationship between the image-capturing camera and the projectionshape, the linking indicating a relationship between positions on theprojection shape and pixels in the captured image; and detect that theposition of the first virtual line of sight is switched to the positionof the second virtual line of sight when a switching occurs of apre-registered virtual line of sight or when a difference betweenunchanged and changed virtual lines of sight is greater than or equal toa specified value.
 2. The vehicle image processing apparatus accordingto claim 1, wherein the processor is configured to change the drawingpart of the predetermined projection shape in the converted image basedon the generated parameter value so that the converted image is a partof the converted image group which is gradually changed from a firstimage viewed along the position of the first virtual line of sight to asecond image viewed along the position of the second virtual line ofsight.
 3. The vehicle image processing apparatus according to claim 1,wherein when the captured image is converted to the image viewed alongat least one virtual line of sight among the first virtual line of sightand the second virtual line of sight, the conversion of the capturedimage is performed without using a predetermined projection shape, andthe virtual line of sight group does not include at least one selectedfrom the first virtual line of sight and the second virtual line ofsight.
 4. A vehicle image processing method comprising: acquiring acaptured image which is an image that has captured an area around avehicle with use of at least one camera; using at least onepredetermined projection shape to convert the captured image into animage viewed along a virtual line of sight that is a line of sightrunning from a predetermined position in a predetermined direction;detecting whether a virtual line of sight is switched from a position ofa first virtual line of sight to a position of a second virtual line ofsight which is a different virtual line of sight, the position of thefirst virtual line of sight associated with a parameter value of atleast one selected from a predetermined position and a direction of thevirtual line of sight and the position of the second virtual line ofsight being associated with a parameter value different from theparameter value associated with the position of the first virtual lineof sight; acquiring at least one type of parameter value associated witha virtual line of sight for each position of the first virtual line ofsight and the second virtual line of sight after detecting that thevirtual line of sight is switched from the position of the first virtualline of sight to the position of the second virtual line of sight, andgenerating a parameter value of a position of a virtual line of sightthat is gradually changed along a predetermined path connecting theposition of the first virtual line of sight and the position of thesecond virtual line of sight; converting the captured image to an imageviewed along at least one virtual line of sight among the first virtualline of sight and the second virtual line of sight using a projectionmap which includes a relationship between a position of a pixel of thecaptured image and a position of a pixel of an image viewed along thefirst virtual line of sight or the second virtual line of sight, andgenerating the converted image to be viewed from the virtual line ofsight included in the virtual line of sight group changing from thefirst virtual line of sight to the second virtual line of sight notusing a projection map but using a predetermined projection shape withthe linking of the position of pixels calculated by the positionalrelationship between the image-capturing camera and the projectionshape, the linking indicating a relationship between positions on theprojection shape and pixels in the captured image; and detecting thatthe position of the first virtual line of sight is switched to theposition of the second virtual line of sight when a switching occurs ofa pre-registered virtual line of sight or when a difference betweenunchanged and changed virtual lines of sight is greater than or equal toa specified value.
 5. The vehicle image processing method according toclaim 4, further comprising changing the drawing part of thepredetermined projection shape in the converted image based on thegenerated parameter value so that the converted image is a part of theconverted image group which is gradually changed from a first imageviewed along the position of the first virtual line of sight to a secondimage viewed along the position of the second virtual line of sight. 6.The vehicle image processing method according to claim 4, wherein whenconverting the captured image to the image viewed along at least onevirtual line of sight among the first virtual line of sight and thesecond virtual line of sight, the conversion of the captured image isperformed without using a predetermined projection shape, and thevirtual line of sight group does not include at least one selected fromthe first virtual line of sight and the second virtual line of sight.